Many modern motor vehicles include electronic sensing mechanisms that try to give the vehicle operator a sense of what is generally behind the vehicle as the vehicle is operated in reverse. For example, injuries may be caused by motor vehicles that are backing up, because the vehicle operator may not see objects in the vehicle path. The potential objects to be avoided, may not have been seen by the vehicle operator because they were in a blind-spot, perhaps obscured by a pillar in the vehicle, or perhaps obscured because they were too low to the operator's field of view. Often such objects are not seen simply because the motor vehicle operator is too preoccupied with reversing the vehicle to pay attention to what is behind the vehicle.
It has been suggested that different types of depth imaging can be used to detect objects around the car. Stereographic camera imaging systems often leave much to be desired in that there is an inherent ambiguity associated reconciling images acquired from two spaced-apart cameras. The depth measurement performance of stereographic cameras degrades rapidly as function of distance. Also, such cameras rely upon brightness information, and can be confused as to distance by bright objects that are farther away from the system than closer objects that reflect less light. Further, stereographic camera imaging systems do not function without ambient light, and thus are of little or no use in dark ambient conditions.
On the other hand, TOF systems can operate without reliance upon brightness data. Some TOF systems emit pulses of infrared optical energy and time how long it takes for emitted pulses to be detected as optical energy that reflects at least partially off a target object. Since the velocity (C) of light is known, the distance Z to a target object is given by Z=t▪C/2, where t is the measured time-of-flight. U.S. Pat. No. 6,323,942 (2001) entitled “CMOS-Compatible Three-Dimensional Image Sensor IC” and assigned to assignee herein Canesta, Inc., describes such a TOF system.
Other TOF systems emit optical energy of a known phase, and determine distances Z by examining phase-shift in the signal reflected from the target object. Exemplary such systems are described in U.S. Pat. No. 6,515,740 (2003) entitled “Methods for CMOS-Compatible Three-Dimensional Imaging Sensing Using Quantum Efficiency Modulation”, or U.S. Pat. No. 6,906,793 (2005) entitled Methods and Devices for Charge Management for Three Dimensional Sensing. These, and other TOF patents, are assigned to assignee herein, Canesta. Inc.
While the present invention operates with various types of depth imaging systems, TOF systems provide especially reliable data, and thus it will be useful to describe briefly a TOF system. FIG. 1A is taken from the '740 and '793 and depicts an exemplary phase-type TOF system. Such systems provide depth information (Z-distance between the sensor and a target object) at each pixel detector in a system detector sensor array for each frame of acquired data. As noted, relative phase shift between emitted optical energy and detected optical energy reflected by a target object is examined to determine Z-range to a target object.
In FIG. 1A, exemplary phase-shift TOF depth imaging system 100 may be fabricated on an IC 110 that includes a two-dimensional array 130 of pixel detectors 140, each of which has dedicated circuitry 150 for processing detection charge output by the associated detector. IC 110 preferably also includes a microprocessor or microcontroller unit 160, memory 170 (which preferably includes random access memory or RAM and read-only memory or ROM), a high speed distributable clock 180, and various computing and input/output (I/O) circuitry 190. Among other functions, controller unit 160 may perform distance to object and object velocity calculations.
Under control of microprocessor 160, optical energy source 120 is periodically energized by an exciter 115, and emits optical energy preferably toward an object target 20. Emitter 120 preferably is at least one LED or laser diode(s) emitting low power (e.g., perhaps 500 mW peak) periodic waveform, producing optical energy emissions of known frequency (perhaps a few dozen MHz) for a time period known as the shutter time (perhaps 10 ms). Typically emitter 120 operates at IR or near IR, with a wavelength of perhaps 800 nm.
Some of the emitted optical energy (denoted S1) will be reflected (denoted S2) off the surface of target object 20. This reflected optical energy S2 will pass through an aperture field stop and lens, collectively 125, and will fall upon two-dimensional array 130 of pixel or photodetectors 140. When reflected optical energy S2 impinges upon photodetectors 140 in array 130, photons within the photodetectors are released, and converted into tiny amounts of detection current. The detection current is typically integrated to accumulate a meaningful detection signal, used to form a depth image.
Thus, responsive to detected reflected optical energy S2 transmitted (as S1) by emitter 120, a three-dimensional image of the visible portion of target object 20 is acquired, from which intensity (A) and Z data can be obtained (DATA). More specifically, reflected incoming optical energy S2 detected by each imaging pixel detector 140 includes intensity information (A), and phase shift information (φ), where phase shift φ varies with distance Z and can be processed to yield Z data. The time-of-flight (TOF) required for optical energy transmitted by emitter 120 to reach target object 20 and be reflected back and detected by pixel detectors 60 is denoted as t. TOF information is captured from which distances Z are determined from the relationship Z1=t▪C/2, where Z is distance to be measured, t is roundtrip TOF time, and C is velocity of light.
TOF sensor system 100 can acquire three-dimensional images of a target object in real time, simultaneously acquiring both luminosity data (e.g., signal amplitude A) and true TOF distance measurements of a target object or scene. FIGS. 1B and 1C depict how a measure of TOF can be determined from shift in phase (φ) between the emitted optical energy (FIG. 1B) and the detected reflected optical energy (FIG. 1C). The DATA obtained from TOF system 100 may be processed to provide video and/or acoustic signals, and/or control system signals. For example, if system 100 were deployed within a motor vehicle and used to detect objects closer than a minimum distance Z, detection of such objects could generate a “Danger” audible command, a “Danger Icon” on a display, and/or a signal that will cause the vehicle to brake or steer in another direction.
Optical energy detected by the two-dimensional imaging sensor array 130 will include amplitude or intensity information, denoted as “A”, as well as phase shift information, denoted as φ. Responsive to pulses or bursts of optical energy transmitted by emitter 120, a three-dimensional image of the visible portion of target object 20 is acquired, from which intensity and Z data is obtained (DATA′). Information within DATA′ may be used to generate an optical display representing target object(s) and their respective distances Z.
FIGS. 2A and 2B depict a prior art object detection and tracking system 200, which includes a prior art depth imaging system 100, which is understood to be a TOF system, a dual stereographic camera system, etc. System 100 is deployed so as to depth image a three-dimensional field of view (FOV) about the imaging system's optical axis 210 generally towards the rear of a motor vehicle 220 moving in reverse along surface 230, as indicated by the heavy arrowed line. In general, the FOV will encompass a desired detection zone, shown as cross-hatched rectangle 240. Detection zone 240 will typically be somewhat wider than the width of vehicle 220, and will extend rearward perhaps 9′ (3 M). As such, a detection zone is defined within which it is desired to recognize objects representing potential objects to vehicle 220, or to recognize objects that could be harmed by vehicle 220.
As indicated in FIGS. 2A and 2B, within detection zone 240 and FOV are a number of real objects, and one phantom object. More specifically, pothole (or the like) 250, tall target object 260-L, and a small target object 260-S. Within the detection zone is what may be reported as a fictitious or phantom object, the inclined region 270 of the roadway itself. Just beyond the detection zone is a medium target object 260-M.
In practice, prior art systems 200 will generally not “see” and thus miss identifying pothole 250 as an object of potential concern. Simply stated, the location along the z-axis of the x-y plane of road 230 is simply not readily known to system 200, and thus identification of the pothole as an object below the plane of the road is not made. The large target object 260L will typically be correctly identified as a potential object but the small target object 260S may often simply not be detected. However object 260S should be detected so a decision can be made whether it may be ignored. Inclined region 270 of roadway 230 is within the detection zone and may be sufficiently high to register as an object of potential concern, even though such identification is spurious, a false-positive. Nonetheless imaging system 100 may actually image or see this inclined surface of the road as a large object of potential concern to vehicle 220. Regretfully false-positives can be dangerous in that they may lull the vehicle operator into simply disregarding all warnings from system 200.
As shown in FIGS. 2B and 2C, within vehicle 220 will be a display 280 or other warning indicator that can advise operator 290 as to the presence of objects of potential concern within the detection zone behind the vehicle. Thus, typically while operating vehicle 220 in reverse, operator 290 could simply look forward at display 280. If system 200 somehow could function perfectly, it would always correctly recognize objects such as 250, 260-L, 260-S, and would reject phantom objects such as inclined road surface 270. If vehicle 220 moved further rearward, object 260M would come into detection zone 240 and would hopefully then be correctly identified as an object of potential concern.
But in practice, as suggested by FIG. 2C, display 280 will correctly depict object 260L, which depiction may be generic rather than an actual image of the object, but will fail to display pothole 250, and small object 260S. Furthermore, display 280 will generally depict as the largest object the inclined roadway portion 270, which indeed to imaging system 100 may look like a large object, but is not. Thus, in addition to not displaying real objects 250 and 260S, prior art system 200 will generate a false-positive and will indicate inclined roadway region 270 as a large object. Upon seeing portrayals of objects of concern on display 280, operator 280 typically will halt vehicle 220 and look rearward or look in the rearview mirror to see what is actually present behind vehicle 220.
Thus, there is a need for an obstacle detection and tracking system useable with depth imaging systems that can identify objects of potential concern, while rejecting false-positive identifications. The class of identifiable objects of potential concern should preferably include potholes and the like, below the average plane of the roadway, as well as small objects that frequently are missed by prior art systems.
The present invention provides such systems and methods for their implementation.